A number of techniques has been proposed to generate a universally focused pan focus image data based upon a plurality of images data by capturing a common scenery at a different focal point. These techniques are generally grouped into “interactive reconstruction methods” and “select and merge methods.” As an example of the select and merge method, Japanese Patent Laid Publication Hei 3-80676 discloses a process where each of a plurality of the input images is divided into blocks and the corresponding blocks are compared for an amount of high frequency portion. Based upon the comparison, the brightness is determined, and an image block containing more of the high frequency portion is selected and merged into a universally focused image. Another example of the select and merge method is Japanese Patent Laid Publication Hei 8-307756, which discloses a technique where a bright portion is selected based upon a change in intensity from a first image that is focused upon an object with a short focal length. The selected bright portion is merged into a corresponding portion of a second image that is focused on another object with a long focal length. In the alternative, an image portion is selected from the first image based upon brightness, and the selected bright portion is merged into the second image at a corresponding position. The selected portion is merged by taking an average of intensity or a change rate in intensity of the corresponding first and second image portions.
Yet another example of the select and merge method in prior art is disclosed in a publication entitled, “Enhanced Image Acquisition by Using Multiple Differently Focused Images” by Naito et al. Electronics and Information Communication Academy D-II Vol. 579-D-II, No. 6, pp 1046-1053 (1996). Each pixel in a first image is compared to a corresponding one in a second image, and the focus of the pixel in the first image is adjusted by applying a predetermined function until it substantially matches with that of the second image. If there is a substantial match, it is assumed that the adjusted pixel of the first image was originally in focus before the adjustment. This process is repeated to determine an in-focus area. The original pixel intensity value is used when the pixel is merged into the universally focused image.
Examples of the interactive reconstruction method includes the publication entitled, “Acquisition of an All-Focused Image by the Use of Multiple Differently Focused Images” by Kodama et al. Electronics and Information Communication, D-II, Vol 80-D-II, No. 9, pp 2298-2307, (1977). The prior art reference discloses a derivation for an equation for generating a universally focused image having multiple focal points at various depths in a scene. The equation is iteratively applied to generate the universally focused image. Similarly, another example, Iterative Generation of Arbitrarily Focused Images by Using Multiple Differently Focused Images. Kodama et al., Image Media Symposium, IMS 96, I-8.18 discloses a technique for generating an image having arbitrarily determined blurness for each depth.
To further illustrate the above iterative reconstruction method by Kodama et al., the following equations are given. It is assumed that a near image and a far image are respectively expressed by f(x) and g(x) while captured images are expressed by I1(x) and I2(x). The whole image I(x) is expressed:                               f          ⁢                                           ⁢                      (            x            )                          =                  {                                                                                                                I                      ⁢                                                                                           ⁢                                              (                        x                        )                                                                                                                        (                                                                        d                          ⁢                                                                                                           ⁢                                                      (                            x                            )                                                                          =                                                  d                          1                                                                    )                                                                                                            0                                                                              (                                                                        d                          ⁢                                                                                                           ⁢                                                      (                            x                            )                                                                          =                                                  d                          2                                                                    )                                                                                  ⁢                                                           ⁢              g              ⁢                                                           ⁢                              (                x                )                                      =                          {                                                                    0                                                                              (                                                                        d                          ⁢                                                                                                           ⁢                                                      (                            x                            )                                                                          =                                                  d                          1                                                                    )                                                                                                                                  I                      ⁢                                                                                           ⁢                                              (                        x                        )                                                                                                                        (                                                                        d                          ⁢                                                                                                           ⁢                                                      (                            x                            )                                                                          -                                                  d                          2                                                                    )                                                                                                                              (        1        )            where the depth of the image is d(x)=d1 or d2I(x)=f(x)+g(x)  (2) then, I1(x)=f(x)+h2g(x)  (3) I2(x)=h1f(x)+g(x)  (4) where h1 and h2 are blur functions. According to this model, equations (2) and (4) lead toG(x)=HI(x)  (5) and G(x)=(h1−1)I1(x)+(h2−1)I2(x)  (6) thus, the following equation is obtained.H=h1h2−1  (7) For each input image, if the blur functions h1 and h2 are known, using the initially reconstructed image I1=I1 or I2.In+1=(H+1)In−G  (8) By iteration, the universally focused image is obtained.
Similarly, an arbitrarily focused image Iab is expressed as follows when blur functions ha and hb are respectively used for a near image f(x) and a far image g(x).Iab(x)=haf(x)+hbg(x)  (9) Instead of Equation (8),Gab=(hbh1−ha)Ii(x)+(hah2−hb)I2(X)  (10) is consideredIn+1=(H+1)In−Gab  (11) for the iterative process for generating the universally focused image. Furthermore, the Kodama et al. reference discloses two ways to determine the blurring function. The first one uses predetermined image capturing conditions including the focal point and at each distance, the blurring function is measured for an object. The second way is that the blurring function is estimated by adjusting the blurness of each pixel as described in the Naito et al. reference.
The above described two groups of prior art techniques are susceptible to errors in generating a universally focused image. In general, the conventional select and merge methods are subject to selection errors, which cause an inferior merged image. The errors are particularly likely near edges where the intensity changes are larger in an out-of-focus image portion than an in-focus image portion.
For the prior-art iterative reconstruction methods, in general, substantially no selection errors are involved. The iterative reconstruction methods do not need information on the location of blurness, and it is possible to generate not only a universally focused image but also an arbitrarily focused image. On the other hand, it is not clear how many iterations are necessary to converge although the Kodama reference discloses only about three iterations. For a large number of iterations, an amount of calculation increases, and it takes a substantial amount of time. In addition, the two proposed techniques for determining the blurring functions also have the following problems. For the first measurement technique, it is necessary to measure the characteristics of the camera as well as other information on each input image such as the focal point length and exposure which affect the blurring function. For these reasons, images taken by an auto focus camera are not usually appropriate for the measurement technique since the auto focus camera generally does not provide information such as the focal point length. The other technique to determine the blurring function based upon the estimation from the image generally requires a large amount of calculation, and it takes a large amount of time. Furthermore, it is difficult for the user to adjust the blurring function.
In addition to the above-described undesirable problems, the prior-art methods also have the following difficulties. Since the zoom ratios usually change as the focal point length is changed, it is necessary to correct at least the zoom ratio of each image before the portions of the images are merged into one composite image. For example, after an auto focus camera is positioned and captures a first image in which a predetermined object is centered, to prepare for capturing a second image, the camera is turned away from the predetermined object and the background scene is focused with the release button. Then, with the above described focus, the camera is turned back towards the predetermined object, and the second image is captured. The second image is centered around the predetermined object. These two images have not only different zoom ratios but also slightly different orientations and positions. The ratio correction alone does not allow the precise positioning of image portions. Images taken by a regular camera without a beam splitter and two CCD's require a high degree of flexible re-positioning technique which includes parallel and rotational movements.
In order to solve the above-described problems, it is desired to quickly generate a high-quality pan focus composite image data from a plurality of common images captured by a generally available auto focus and auto exposure camera. To accomplish the above objectives, it is desired 1) to speed up the image composition process based upon the use of a blurring function, 2) to speed up the blurring function determination process, 3) to facilitate the confirmation and correction of the results by the blurring function, 4) to speculate the relative position of image data and to enable the position match for the difficult image data that is corrected for the zoom ratio, and 5) to facilitate the confirmation and the correction of the image data and the relative positioning results.